Create Review Passes and Batches

Review passes are based on searches saved in OPEN DISCOVERY. There are two modes of review passes: Active Learning disabled and Active Learning enabled. With Active Learning disabled review passes, searches are planned and saved in OPEN DISCOVERY Review before you create the review passes. With Active Learning enabled review passes, you must scope the documents by using either Entire Case, Saved Search, or Custom Index. Coding forms are also required for a review pass. For a detailed overview, see Review Passes and Batches.

Note: Active Learning disabled review passes still run Active Learning “in the background”, but it is transparent to reviewers. At some point in the review, the setting could be enabled (Active Learning enabled), to start prioritizing predicted positive documents within the review pass, and that would change the document order for reviewers.

If you prefer to turn off the Active Learning functionality running “in the background” on Active Learning disabled review passes, the functionality can be turned off across the environment through SQL. For more information, please contact IPRO Support.

ClosedCreate Review Passes

You can create either an Active Learning enabled review pass or an Active Learning disabled review pass.

What is Active Learning?

Active learning is the ability to have documents prioritized through the use of algorithms with no direction from the user beyond the coding supplied when they review the documents. With Active Learning, the prioritized documents are more likely to be relevant to the review purpose selected.

The “continuous” facet of Active Learning relates to the process of ranking and re-ranking documents for manual review based on the constant stream of incoming coding throughout the review lifecycle. That is, Active Learning reflects not only an initial set of assessments to rank and prioritize documents, but continuously updates those rankings based on the most recent assessments.

Review passes are designed for Active Learning to be enabled or disabled. When creating a review pass, Active Learning-enabled is the default.

Create an Active Learning Enabled Review Pass

To create an Active Learning enabled review pass:

  1. In OPEN DISCOVERY Review, click a case card.

  2. Click the button.

    The Visual Search Dashboard appears.

  3. On the left panel, click the Review Pass Management button, .

  4. Click . In Review Pass Details, make sure the Turn off Active Learning switch remains in the “on” position as Active Learning is on by default.

  5. Enter a name for the review pass and complete other required entries. Continue setting up the AL enabled review pass by using the definitions in the tables accessed by clicking ClosedReview Pass Definitions - AL Enabled.

The following tables describe the definitions shown on the Settings tab and the Overview tab.

Settings Tab (AL Enabled)

SETTINGS

DESCRIPTION

General

Active Learning

Switch on - enabled

Switch off - disabled
Coding Form

Coding form being used for the review pass.

Note: A unique coding form must be created for each Active Learning Enabled review pass.
Primary Review Purpose

Allows IPRO Analytics to learn from the work done by reviewers. If the Primary Review Purpose tag is applied to a document, then the system will learn from that document.

  • If the document is marked "Reviewed" and the Primary Review Purpose is applied, then that document is recorded as a "positive"

  • If the document is marked "Reviewed" and the Primary Review Purpose is not applied, then that document is recorded as a "negative"

Reviewers should use the On Hold review status when reviewing documents if they are unsure whether the document is a positive or negative example. This prevents the document from being added as an example until a review decision can be made.

Note: A unique "Responsive Tag" must be created for each Active Learning Enabled review pass.

Recall

The estimate of the percentage of positive documents in the entire case. The desired recall for the review pass can be configured from 70% to 90% in increments of 10%.

Batch Size

The number of documents a reviewer will receive at a time

Layout Style

Determines the view when the review pass is opened

Document Scope

The scope of documents that Active Learning will use train documents. Select from:

  • Entire Case (recommended - Active Learning works best when the entire case is used)

  • Saved Search - (due to the nature of Active Learning, often it is best to not filter the content submitted to a project, but Saved Search can be used if necessary)

  • Custom Index - (intended for advanced users. Allows the user to create an index in OPEN DISCOVERY ahead of time. This gives them additional options for configuration, and then the index can be selected for use in the Review Pass.

Permissions

Group(s)

Contains Groups assigned to the normal batches in the review pass

Review QC Groups(s)

Controls the QC batch permissions. Contains Groups assigned to the QC batches within the review pass.

Note: Not selecting either of these group types allows access to both types of groups.

If a Review QC Group is selected, users who are part of that group will always receive a QC Batch for the Review pass if one is available. If a QC Batch is not available and the user is part of the regular review group, they will receive a normal batch.

Overview Tab (AL Enabled)

Sections

DESCRIPTION
Review Metrics Breakdown of selected tags in checked-in documents

Precision is the percent likelihood that you will hit a positive document as you continue reviewing. For example, if you review 100 documents and 90 of them are positive, then the Precision is 90%.

Review Progress

Manually reviewed checked-in documents by date

Documents Manually Reviewed Listing of manually reviewed documents

Batch Information

Batch details for the review pass

Create an Active Learning Disabled Review Pass

To create a Active Learning-disabled review pass:

  1. Click . In Review Pass Details, make sure Turn off Active Learning is switched to the “off” position as Active Learning is on by default.

  2. Enter a name for the review pass and complete other required entries.

  3. As required, complete optional entries.

  4. Go to Create Batches to continue defining the review pass.

  5. After you have created the batches, continue setting up the AL disabled review pass as defined in the tables accessed by clicking this link: ClosedReview Pass Definitions - AL Disabled.

The following tables describe the definitions shown on the Settings tab and the Overview tab.

SETTINGS

DESCRIPTION

General

Active Learning

Switch on - enabled

Switch off - disabled
Coding Form Coding form being used for the review pass.

Primary Review Purpose

Allows IPRO Analytics to learn from the work done by reviewers. If the Primary Review Purpose tag is applied to a document, then the system will learn from that document.

  • If the document is marked "Reviewed" and the Primary Review Purpose is applied, then that document is recorded as a "positive"

  • If the document is marked "Reviewed" and the Primary Review Purpose is not applied, then that document is recorded as a "negative"

Reviewers should use the On Hold review status when reviewing documents if they are unsure whether the document is a positive or negative example. This prevents the document from being added as an example until a review decision can be made.

Recall

The estimate of the percentage of positive documents in the review pass. The desired recall for the review pass can be configured from 70% to 90% in increments of 10%.

Saved Search

Saved search of records to be included in the review pass.

Batch Size

The number of documents a reviewer will receive at a time.

Batch Prefix

The prefix added before each batch.

Batch By

Method for grouping documents within batches. Ensure None is selected if you prefer to organize batches in numerical order by image key. Otherwise, you can choose to batch documents either by field or by analytics categories.

For example, you might group documents based on the File Type field, in which case all like documents (.MSG, .DOC, .XLS, etc.) will be grouped into batches based on batch size.

Batch Priority

The priority set for each batch.

Family Field

Ensures all documents with the same value in this field are kept together in batches.

Layout Style

Determines whether the reviewer can see only the Document Viewer (Form View) or both the Document Viewer and the Grid (Grid View).

Batching Order

Sort documents by the content of up to three fields.

Permissions

Group(s)

Contains Groups assigned to the normal batches in the review pass.

Review QC Groups(s)

Controls the QC batch permissions. Contains Groups assigned to the QC batches within the review pass.

Note: Not selecting either of these group types allows access to both types of groups.

If a Review QC Group is selected, users who are part of that group will always receive a QC Batch for the review pass if one is available. If a QC Batch is not available and the user is part of the regular review group, they will receive a normal batch.

ClosedCreate Batches

To create batches:

  1. In the Review Pass Management panel, click the required review pass. If a large number of review passes exists, take any of the following actions to find the required one:

    1. Scroll vertically to locate the review pass in the list.

    2. Sort the keyword list names by clicking the sort button, , and selecting from the different options. Click a selected option again to reverse the sort order.

    3. In the Search box , start typing any of the following items: the keyword list name, the name of the administrator who created/modified the group, or part of the created/modified date. This is a plain text search that searches these items in order.

  2. Click Update Documents and Create Batches if it is required for unbatched documents to be batched. For details, see Update Documents.

  3. Review the number of unbatched documents in the review pass, then click Create Batches.

  4. Repeat the previous steps to create other review passes.

Update Documents

To batch unbatched documents for the selected review pass:

  1. Click Update Documents and Create Batches. The following dialog box appears:

    Click Continue. A message displays to let you know the updated batch is being built.

  2. View the Batch Information section to see the updated batch status.

ClosedRe-batch Unreviewed Documents

To re-batch unreviewed documents:

  1. If it is required that you create a new review pass, first complete the Create Review Passes and Batches section.
  2. Click Re-Batch Unreviewed Docs.
  3. After it is determined in the Review Pass Information area that there are unreviewed documents, click Re-Batch Unreviewed Docs to have these documents re-batched into new batches.

    		4.

    Note: The Re-Batch Unreviewed Docs button also re-batches On Hold Batches.

 

Related Topics

Manage Review Passes

Manage Batches

Active Learning Frequently Asked Questions